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It is a key to construct a similarity graph in graph-oriented subspace 
learning and clustering. In a similarity graph, each vertex denotes 
a data point and the edge weight represents the similarity between 
two points. There are two popular schemes to construct a similarity 
graph, i.e., pairwise distance based scheme and linear representation 
based scheme. Most existing works have only involved one of the 
above schemes and suffered from some limitations. Specifically, pairwise 
distance based methods are sensitive to the noises and outliers compared 
with linear representation based methods. On the other hand, there 
is the possibility that linear representation based algorithms wrongly 
select inter- subspaces points to represent a point, which will degrade 
the performance. In this paper, we propose an algorithm, called Locally 
Linear Representation (LLR), which integrates pairwise distance with 
linear representation together to address the problems. The proposed 
algorithm can automatically encode each data point over a set of points 
that not only could denote the objective point with less residual error, but 
also are close to the point in Euclidean space. The experimental results 
pf^ show that our approach is promising in subspace learning and subspace 

1 ^ clustering. 

o' 

^introduction: As a useful technique in subspace learning and clustering 

5_^ethods, graph-oriented algorithm has become a hot topic in machine 

Ofj ^minp and computer vision, whose key is to construct a similarity 

,^^faph (T]. In the graph, each vertex denotes a data point and the edge 

vveight represents the similarity between the connected points. Currently, 

■^jlhere are two ways to calculate the similarity among data points |2|, i.e., 

^Npairwise Distance based Scheme (PDS) and Linear Representation based 

Scheme (LRS). PDS measures the similarity by using the distance between 

I ytwo points, e.g., Laplacian Eigenmaps (LE) |3| adopts the Euclidean 

^^distance with heat kernel to build the similarity graph. Alternatively, LRS 

, A ssumes that each data point could be denoted as a linear combination 

of some intra-subspace points |6). Thus, the linear representation 

^'Coefficients could be regarded as a kind of similarity. Recently, LRS 

l_jTjas attracted a lot of interests from subspace learning and clustering 

community since it could be better to capture the real data structure. For 

,_pxample. Locally Linear Embedding (LLE) |4|, Neighborhood preserving 

^embedding(NPE) |5|, Sparse Subspace Clustering (SSC) |6|, and Low 

(^Jlank Representation (LRR) |7|. 

QQ It is notable that these algorithms have only involved one of the 
^|*chemes, and suffered from some limitations. Particularly, pairwise 
VO(iistance based methods are sensitive to the noises and outliers compared 
.w ith linear representation based methods, since the pairwise distance only 
\l depends on the two connected data points and ignores the global stmcture 
>— <)f the whole data set. On the other hand, the subspaces spanned by the 
^^tiata points are usually dependent. Therefore, it is probable that linear 
^"representation based algorithms will wrongly select inter-subspaces points 

^o represent a point, and result in a degraded performance. 

. i-H This letter proposes an algorithm, named as Locally Linear 

^Oiepresentation (LLR), which incorporates pairwise distance into linear 

S^epresentation. It selects the points that not only could denote the objective 

C^Doint with less residual error, but also are close to the point in terms of 

Euclidean distance. Moreover, the solution of the proposed method can be 

analytic, which forms another advantage of LLR. 



Locally Linear Representation (LLR): Our basic idea derives from a 
widely-accepted assumption in manifold learning that a topological 
manifold is a topological space that is locally homeomorphic to an 
Euclidean space. It implies that the intra-subspace points not only can 
linearly represent each other, but also are close to each other in terms 
of Euclidean distance. However, most previous works only focused on 
one aspect of them and accordingly developed linear representation based 
algorithms and pairwise distance based algorithms. As discussed in Section 
I, these methods encountered some problems. 

Fig.[T]gives a toy example to illustrate the limitations of PDS and LRS 
and show the effectiveness of our approach. It is clear that either PDS 
or LRS will wrongly find the nearest neighbors for point A, while our 
approach will get the correct answer by selecting the points not only belong 




(a) PDS 



(b) LRS 



(c) Our method 



Fig. 1 The geometric analysis of three different similarity graph construction 
strategies. There are three subspaces Si, S2, and S3 lie in R^, where 
dim{S\) = 1, dim{S2) = 1, dim^S^) = 2. Points A, C, D are draw from 
S3, point B from S2, and point E from Si. Fig. \l(a)\ shows that the nearest 
neighbor of A is B in terms of Euclidean distance; Fig. \l(b)\ shows that the 
nearest neighbor of A is B and E in terms of linear representation based 
similarity; Fig. \l(c)\ shows that our method will select C and D as the nearest 
neighbor of A. 



to the same subspace with A, but also are close with A in terms of pairwise 
distance. 

The objective function of our proposed approach is defined as 
following: 



i^A||SiCi||-H(l-A)|| 



DiCi 



s.t. 



I'ci 



(1) 



where c^ 6 M" is the similarity of a data point x^ g R™ over the 
corresponding dictionary D^ = [xi,X2, . . . ,Xi_i,0, x^-i-i, . . . , x„]. Si is 
a diagonal matrix whose j*'' diagonal element is the pairwise distance from 
Xi to the j"' point in Di, (j = 1, 2, . . . , n), 1 £ M™ is a vector composed 
with ones, and A g [0, 1) is the trade off between pairwise distance strategy 
and linear representation strategy. 

By solving the equation (T), it gives that 






(2) 



where M, = ASf S, -(- (1 - A)(xa^ - D,)^(xa^ - D,). 

However, the above solution is not sparse which produces too many 
connections inside the graph. It results in a high computing cost in 
subspace learning and clustering and decreases the discrimination between 
different subspaces. In this letter, we can get a more discriminate similarity 
matrix C by preserving the k strongest connections for each data point as 
in Q. In other words, we only keep k largest entries in Ci, and set others 
to zeros. Consequently, we get a sparse similarity graph which has high 
discrimination. 

Given a data setXeK"*^", the proposed LLR algorithm constructs a 
similarity graph via the following steps: 

(1) For each point Xi(i = 1, 2, . . . , n), calculate its similarities, denoted 
as Ci, to other points via (|2). 

(2) Get a more discriminated representation Ci by keeping the k- 
strongest connected points to point Xi by using fc-nearest neighbor 
searching on Ci , and set all other elements in Ci as zeros, where i = 
1,2, ...,n. 

(3) Construct a similarity graph by treating each point as a vertex, 
and assigning the connection weight between the node i and j as Wij = 

l^ij I + l^ji I- 

Once the similarity graph is built, multiple existing algorithms can be 
integrated with this similarity graph for subspace learning and subspace 
clustering. Subspace learning aims to find a projection matrix P g Jj^xd 
for transforming the high-dimensional datum into a low-dimensional 
space. Since the similarity graph is preserved in the transformation, 
following the embedding step of NPE |5j, LLR graph can be used to 
develop a subspace learning algorithm. Then, the classification process 
can produce on the low-dimensional representation with much lower 
computational cost. Furthermore, subspace clustering refers to the problem 
of separating the data points according to their underlying subspaces J6). 
The spectral clustering is the most popular algoiithm for this task, but there 
is a challenge to measure the similarity between any two data points. We 
can use the the LLR graph as the similarity graph for spectral clustering. 
Similar to SSC 1 1. 6, 7 1, we can easily conduct spectral clustering via LLR 
graph, and achieve satisfactory performance, which will be shown by the 
simulation results presented in the next section. 

Experimental verification: To verify the effectiveness of LLR in subspace 
learning and clustering tasks, we carried out the experiments over two 
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Table 1: Accuracy of 1-NN classifier for different subspace learning 
algorithms on AR database and Extended Yale Database B 



AR 
ExYaleB 



0.952 
0.953 



0.734 
0.848 



0.881 
0.925 



Table 2: Clustering quality (Accuracy (AC) and Normalized Mutual 
Information (NMI)) of spectral clustering algorithm based on the graph 
built by LLR, LRR, SSC, LLE, and NSC. 



Database 


Metric 


LLR 


LRR 


SSC 


LLE-Gi-aph 


NSC 


AR 


AC 

NMI 


0.837 
0.929 


0.771 
0.910 


0.767 
0.886 


0.396 
0.682 


0.361 
0.652 


ExYaleB 


AC 

NMI 


0.883 
0.922 


0.713 
0.772 


0.613 
0.684 


0.461 
0.540 


0.426 
0.539 



popular databases, i.e., Extended Yale database B 1 8 1 and AR database 1 9 1. 
The used Extended Yale database B contains 2014 near frontal face images 
of 38 individuals (58 images for each individual), where each image is 
downsized from 192 x 168 to 48 x 42. The used AR database contains 
1400 face images without disguises distributed over 100 individuals (14 
images for each subject), where each image is cropped from 165 x 120 
to 55 X 40. For computational efficiency, we perform PCA to reduce the 
dimensionality of data by retaining 98% energy of the data. Moreover, to 
speed up LLR, we use 300 nearest neighbors as dictionary for each point. 
Similar to QHHT], in the all experiments, we reported the best result of 
the tested algorithms with tuned parameters. 

We firstly investigate the performance of LLR in subspace learning 
by comparing with Neighborhood Preserving Embedding (NPE) Q 
and Locality Preserving Projection (LPP) |10|. In the experiments, we 
randomly select a half of images from each subject (AR and Extended 
Yale B) for training purpose, and use the remaining as testing data. 
Table 1 reports the classification accuracy of fc-Nearest Neighbor Classifier 
(k = 1, i.e., 1-NN) with the evaluated algorithms. From the results, we 
could see that the proposed algorithm outperforms the other methods with 
considerable performance margin. For example, the classification accuracy 
of LLR is at least higher 7.1% than the other tested methods over AR, 
and the figure is about 2.8% over Extended Yale Database B. Note that, 
NPE constructs a similarity graph based on linear representation, while 
LPP obtains the graph by computing the distance between two data points. 

As discussed in the Section II, the graph built by LLR could be used 
for subspace clustering in the frainework of spectral clustering. In this 
part, LLR was compared with several state-of-the-art algorithms. LRR tZJ, 
SSC (6), LLE-graph, and Normalized Spectral Clustering (NSC) fTTl . 
Two popular metrics. Accuracy (AC) and Normalized Mutual Information 
(NMI) 1 1 1, are used to evaluate the clustering quality. 

Table 2 reports the clustering results of LLR, from which we could have 
the following observations: 

• LLR outperforms the other methods in AC and NMI with distinct 
advantage. For example, LLR is 6.6% and 1.9% higher than the second 
best method (LRR) over AR in AC and NMI, respectively. The figures 
are 17% and 15% over ExYaleB. 

• LRR and SSC are two recently proposed algorithms, which are 
superior to LLE-graph and NSC which respectively are based on linear 
representation and pairwise distance (Euclidean distance + heat kernel). 

Conclusion: Linear representation and pairwise distance are two popular 
ways to construct a similarity graph for subspace learning and clustering, 
which derive from one aspect of the assumption used in manifold learning 
such that they suffer from some limitations. To enhance the connectedness 
among intra-subspace points and disconnectedness among intra-subspace 
points, we proposed an algorithm to construct a similarity graph by 
enforcing the selected points are close not only in Euclidean distance but 
also could reconstruct the objective point with less residual error. Extensive 
experiments verified the effectiveness of our approach and corroborated the 
claims that combination of linear representation and pairwise distance are 
helpful to improve the discrimination of data. 
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